The present invention relates to an image synthesizing apparatus structured such that overlap portions are provided for each image, followed by dividing one composition into sections to be photographed by a plurality of photographing operations and followed by sequentially connecting the photographed images to one another so that a precise image or an image having a wide angle of view is formed or an all round panoramic photograph is taken.
Recently, reduction in the cost of high-performance personal computers has been realized, causing personal computers to widely be used in companies, schools and homes. Simultaneously, visual information apparatuses including digital still cameras and digital video cameras have been widely used because of easiness of capturing an image into the personal computer in place of conventional film cameras.
The digital camera is not required to perform a development process of a photographed image and is permitted to easily delete and edit photographed images. Therefore, digital cameras have widely been used mainly by users of the personal computers.
However, images photographed by a solid-state image device, such as CCD, suffer from inferior resolution to that of an image exposed to a film. Therefore, raising of the resolution has considerably be required.
The resolution can be raised by increasing the number of pixels of the image pickup device. A fact is known that the cost of the solid-state image device is excessively raised in proportion to the number of pixels. Thus, the overall cost of the camera is raised excessively.
Therefore, the applicant of the present invention has developed a technique for connecting, to one another, images photographed by a plurality of image pickup devices and another technique of photographing a subject in a division manner by, for example, moving a camera, to connect images photographed by one image pickup device, which has been filed as U.S. patent application Ser. No. 08/969,937.
An example of the image synthesizing techniques of a type in which a plurality of image pickup devices are disposed so as to be operated as one image pickup device will simply be described with reference to FIG. 18A.
The foregoing image pickup apparatus incorporates image input portions 1a, 1b and 1c each having an optical system 2, an image pickup portion 3 including an image pickup device, such as a CCD, an A/D conversion portion 4 for converting analog data to digital data; an image synthesizing portion 6 for connecting the supplied images A, B and C; a monitor 7 for displaying a synthesized image; a printer 8; and a storage medium 9. Although each of the image input portions 1a, 1b and 1c generally incorporates an image processing portion for adjusting white balance for compressing image data so as to efficiently store image data, the image processing portion is omitted from illustration.
The image of a subject 5 is divided into a plurality of images each having overlap portions, and then photographed by the image input portions 1a, 1b and 1c. The photographed images are supplied to the image synthesizing portion 6. The image synthesizing portion 6 uses image data of the overlap portions of the images to calculate the positional relationship among the images. Then, the image synthesizing portion 6 forms and transmits a connected image to the monitor 7 or the like.
FIG. 23 shows the structure of the image synthesizing portion 6.
The images A, B and C photographed by the image input portions 1a, 1b and 1c are temporarily stored in frame memories 10a, 10b and 10c. Then, deviation detectors 11a and 11b calculate the positional relationships (amounts of parallel translation and quantities of rotations) of the adjacent images in accordance with image data of the overlap portions. The results of the calculations are supplied to interpolators 12a and 12b. 
The interpolator 12a interpolates the image B by using the output of the detector 11a, generating an interpolated image B. The interpolator 12b interpolates the image C by using the outputs of the detectors 11a and 11b, generating an interpolated image C. The image A used as a reference image in this case, the interpolated image B, and the interpolated image C are supplied to a coefficient determining device 13. The pixels of the image A, those of the interpolated image B, and those of the interpolated image C are multiplied by a predetermined coefficient used in weighting addition. The three images subjected to the multiplication are supplied to an adder 15, which adds the correct images, forming a combined image. The combined image is temporarily stored in a frame memory 16 and can be output therefrom whenever necessary.
As shown in FIG. 18B, the coefficient determining device 13 determines coefficients Ca, Cb and Cc in such a manner that the weights are gradually reduced (or enlarged) in the overlap portions.
The above-mentioned technique is able to effectively raise the resolution of a digital image. Moreover, the technique can be used to enlarge the angle of view. In addition, the technique can be applied to the following technique as well as the technique for simply enlarging the angle of view.
Software has been put into practical use so that an image photographed in an angular degree of 360xc2x0 about a certain point, that is, a so-called panoramic image is used as an original image. Then, a required viewpoint is selected by operating a mouse or a keyboard to form an image photographed by a camera directed to the direction of the selected viewpoint. Thus, the image is displayed on a display unit. The foregoing software enables a user to arbitrarily observe subjects from a required direction. Therefore, the user is able to feel a virtual reality as if the user standing at the photographing point looks around the user.
To form the panoramic image, two methods have been widely used.
One of the two methods is a method using a panoramic camera having an aperture portion which is rotated in synchronization with the operation for winding a film up so that an image is photographed. Another method is a method of forming a panoramic image by rotating a usual camera on a horizontal plane to photograph divided sections of a subject by performing a plurality of photographing operations and by connecting the formed images to one another.
The latter method of the two methods will now be described.
The latter method uses the technique disclosed in the ITEC ""91 theses, ITE Annual Convention, 1991, and shown in FIGS. 19 and 20.
Referring to FIG. 19, symbol R represents subject and O represents a reference point of a camera (in general which is a node of an optical system). Projection planes Q and Qxe2x80x2 are planes onto which the subject is projected when the subject is photographed. In the foregoing case, a cylinder P is a projection plane equivalent to the projection plane when a panoramic image is photographed by the first method, the cylinder P being a cylinder formed around the point O.
A plurality of images projected onto the projection planes Q and Qxe2x80x2 are converted into images projected onto the cylinder P, followed by connecting adjacent images to one another.
For example, FIG. 20A shows images photographed by a camera and projected onto the projection planes Q and Qxe2x80x2. FIG. 20B shows images formed by projecting photographed images from the projection planes Q and Qxe2x80x2 to the cylinder P. When the foregoing images are connected to each other by a method disclosed in U.S. Ser. No. 08/969,937 an image as shown in FIG. 20C can be obtained. The obtained image is equivalent to the image photographed by the former method of the two methods.
The technique is structured to perform a smoothing process when the adjacent images are connected to each other. Therefore, the overlap portions can smoothly be connected to each other. If a subject as shown in FIG. 21 is photographed, the sky having a large ratio is photographed in the left-hand image, as shown in FIG. 22A. As compared with a right-hand image shown in FIG. 22B in which only a building is photographed, the overall image is therefore exposed relatively dark. When the two images are connected to each other by the above-mentioned disclosed technique, the difference in the exposure can be smoothed in the connected image. However, the difference in the brightness becomes conspicuous between the right-hand portion and the left-hand portion. As a result, an unnatural image is undesirably formed.
In Jpn. Pat. Appln. KOKAI Publication No. 9-93430, a technique for correcting the exposure after which images are connected to one another has been disclosed. According to the foregoing disclosure, a method has been disclosed in which the difference in the average value between the overall overlap portions of the images is used to determine a correction coefficient. Another method has been disclosed in which a plurality of feature points are detected so that a correction coefficient is determined from the pixel values of the detected feature points by a least-square method.
If registration is appropriately performed by the conventional technique when the images are connected to each other by the above-mentioned technique for synthesizing images, exposure correction has the following problem. That is, when the method disclosed in Jpn. Pat. Appln. KOKAI Publication No. 9-93430 is employed to correct the exposure, the portion having a pixel value near the average value can substantially accurately be connected. However, pixel values different from the average value have large errors. Since a plurality of feature points are searched for, excessively long time is required to complete the process. Thus, the foregoing problems arise.
The technique disclosed in Jpn. Pat. Appln. KOKAI Publication No. 9-93430 has the structure that the values allowed to overflow the gradation range of the original image are normalized after the correction has been performed to maintain the same number of gradient levels as that of the original image before it is corrected. However, the above-mentioned method encounters a false contour which is formed in a portion having a highest gradation value.
To employ the latter method for forming a panoramic image when the images are connected to one another, the angles of view of the photographing devices must be known in order to determine the radius of the cylinder onto which the photographed image is projected. Although the angle of view can be calculated from the focal distance and the size of the image pickup device, many users do not have the foregoing knowledge. In recent years, cameras of a type having a lens provided with a zooming function have been widely used. The focal distance realized at the photographing operation can be detected only at the two ends of the zooming movement.
An object of the present invention is to provide an image synthesizing apparatus arranged to connect and synthesize a plurality of images obtained by photographing one subject (one composition) to have overlap portions (overlap regions) so as to form one image and capable of appropriately connecting the images to one another by automatically calculating angles of view from the overlap portions of adjacent images so as to correct the angles of view and accurately correcting exposure in accordance with pixel values of the overlap portions while elongation of processing time is prevented.
According to one aspect of the present invention, there is provided an image synthesizing apparatus for connecting divided images obtained by photographing one composition such that the one composition is divided into a plurality of sections in such a manner that overlap portions between adjacent images are included so that the original composition is restored, the image synthesizing apparatus comprising: exposure correction means for selecting at least a portion of pixel groups in the overlap regions of the divided images, the positional relationship of each of which has been corrected such that connection of adjacent images is permitted, calculating a correction coefficient which is determined in accordance with an average and dispersion of pixel values and correcting exposure of each of all images which must be connected to one another or each images except for a reference image in accordance with the correction coefficient; and image synthesizing means for producing an image of the original composition by connecting the divided images having the exposure corrected by the exposure correction means.
According to another aspect of the present invention, there is provided an image synthesizing apparatus for connecting and synthesizing divided images obtained by photographing one composition such that the one composition is divided into a plurality of sections in such a manner that overlap portions between adjacent images are included, the image synthesizing apparatus comprising: image correction means structured to correct the positional relationship such that the connection of adjacent images is permitted by setting at least one portion of the same subject included in each of the overlap portions of adjacent images of the divided images as specific points and by subjecting data of coordinates of the positions of the specific points to a comparison to estimate angles of view of the images so as to perform a geometrical deforming process in accordance with estimated angles of view such that the positional relationships of the images coincide with each other; and image synthesizing means for connecting and synthesizing the divided images processed by the image correcting means.
The image synthesizing apparatuses according to the present invention have the structure to form a panoramic image (an image having a wide angle of view) by connecting a plurality of divided images obtained by photographing one subject or one composition such that the subject or the composition is divided into sections by the steps of selecting at least a portion of pixel groups of the overlap regions before the images are connected and synthesized; calculating a correction coefficient determined in accordance with an average pixel value and a dispersion value; subjecting the overall image which must be connected to one another or images except for a reference image to an exposure process to correct the exposure in accordance with the correction coefficient; and sequentially connecting and synthesizing the images each having the corrected exposure so that a panoramic image connected naturally in a viewpoint of the exposure is formed.
Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter.